ROAM: a Rich Object Appearance Model with Application to Rotoscoping

Rotoscoping, the detailed delineation of scene elements through a video shot, is a painstaking task of tremendous importance in professional post-production pipelines. While pixel-wise segmentation techniques can help for this task, professional rotoscoping tools rely on parametric curves that offer the artists a much better interactive control on the definition, editing and manipulation of the segments of interest. Sticking to this prevalent rotoscoping paradigm, we propose a novel framework to capture and track the visual aspect of an arbitrary object in a scene, given a first closed outline of this object. This model combines a collection of local foreground/background appearance models spread along the outline, a global appearance model of the enclosed object and a set of distinctive foreground landmarks. The structure of this rich appearance model allows simple initialization, efficient iterative optimization with exact minimization at each step, and on-line adaptation in videos. We demonstrate qualitatively and quantitatively the merit of this framework through comparisons with tools based on either dynamic segmentation with a closed curve or pixel-wise binary labelling

Learning how to be robust: Deep polynomial regression

Polynomial regression is a recurrent problem with a large number of applications. In computer vision it often appears in motion analysis. Whatever the application, standard methods for regression of polynomial models tend to deliver biased results when the input data is heavily contaminated by outliers. Moreover, the problem is even harder when outliers have strong structure. Departing from problem-tailored heuristics for robust estimation of parametric models, we explore deep convolutional neural networks. Our work aims to find a generic approach for training deep regression models without the explicit need of supervised annotation. We bypass the need for a tailored loss function on the regression parameters by attaching to our model a differentiable hard-wired decoder corresponding to the polynomial operation at hand. We demonstrate the value of our findings by comparing with standard robust regression methods. Furthermore, we demonstrate how to use such models for a real computer vision problem, i.e., video stabilization. The qualitative and quantitative experiments show that neural networks are able to learn robustness for general polynomial regression, with results that well overpass scores of traditional robust estimation methods

Discovering motion hierarchies via tree-structured coding of trajectories

International audienceThe dynamic content of physical scenes is largely compositional, that is, the movements of the objects and of their parts are hierarchically organised and relate through composition along this hierarchy. This structure also prevails in the apparent 2D motion that a video captures. Accessing this visual motion hierarchy is important to get a better understanding of dynamic scenes and is useful for video manipulation. We propose to capture it through learned, tree-structured sparse coding of point trajectories. We leverage this new representation within an unsupervised clustering scheme to partition hierarchically the trajectories into meaningful groups. We show through experiments on motion capture data that our model is able to extract moving segments along with their organisation. We also present competitive results on the task of segmenting objects in video sequences from trajectories

Hierarchical Motion Decomposition for Dynamic Scene Parsing

Peer-reviewed paper accepted for presentation at the IEEE International Conference on Image Processing 2016International audienceA number of applications in video analysis rely on a per-frame motion segmentation of the scene as key preprocess-ing step. Moreover, different settings in video production require extracting segmentation masks of multiple moving objects and object parts in a hierarchical fashion. In order to tackle this problem, we propose to analyze and exploit the compositional structure of scene motion to provide a segmen-tation which is not purely driven by local image information. Specifically, we leverage a hierarchical motion-based partition of the scene to capture a mid-level understanding of the dynamic video content. We present experimental results showing the strengths of this approach in comparison to current video segmentation approaches

Human-robot interaction torque estimation methods for a lower limb rehabilitation robotic system with uncertainties

Lower limb rehabilitation robot (LLRR) users, to successfully conduct isotonic exercises, require real-time feedback on the torque they exert on the robot to meet the goal of the treatment. Still, direct torque measuring is expensive, and indirect encoder-based estimation strategies, such as inverse dynamics (ID) and Nonlinear Disturbance Observers (NDO), are sensitive to Body Segment Inertial Parameters (BSIPs) uncertainties. We envision a way to minimize such parametric uncertainties. This paper proposes two human–robot interaction torque estimation methods: the Identified ID-based method (IID) and the Identified NDO-based method (INDO). Evaluating in simulation the proposal to apply, in each rehabilitation session, a sequential two-phase method: (1) An initial calibration phase will use an online parameter estimation to reduce sensitivity to BSIPs uncertainties. (2) The torque estimation phase uses the estimated parameters to obtain a better result. We conducted simulations under signal-to-noise ratio (SNR) = 40 dB and 20% BSIPs uncertainties. In addition, we compared the effectiveness with two of the best methods reported in the literature via simulation. Both proposed methods obtained the best Coefficient of Correlation, Mean Absolute Error, and Root Mean Squared Error compared to the benchmarks. Moreover, the IID and INDO fulfilled more than 72.2% and 88.9% of the requirements, respectively. In contrast, both methods reported in the literature only accomplish 27.8% and 33.3% of the requirements when using simulations under noise and BSIPs uncertainties. Therefore, this paper extends two methods reported in the literature and copes with BSIPs uncertainties without using additional sensors

Relevance of gastrointestinal manifestations in a large Spanish cohort of patients with systemic lupus erythematosus: what do we know?

SLE can affect any part of the gastrointestinal (GI) tract. GI symptoms are reported to occur in >50% of SLE patients. To describe the GI manifestations of SLE in the RELESSER (Registry of SLE Patients of the Spanish Society of Rheumatology) cohort and to determine whether these are associated with a more severe disease, damage accrual and a worse prognosis. METHODS: We conducted a nationwide, retrospective, multicentre, cross-sectional cohort study of 3658 SLE patients who fulfil =4 ACR-97 criteria. Data on demographics, disease characteristics, activity (SLEDAI-2K or BILAG), damage (SLICC/ACR/DI) and therapies were collected. Demographic and clinical characteristics were compared between lupus patients with and without GI damage to establish whether GI damage is associated with a more severe disease. RESULTS: From 3654 lupus patients, 3.7% developed GI damage. Patients in this group (group 1) were older, they had longer disease duration, and were more likely to have vasculitis, renal disease and serositis than patients without GI damage (group 2). Hospitalizations and mortality were significantly higher in group 1. Patients in group 1 had higher modified SDI (SLICC Damage Index). The presence of oral ulcers reduced the risk of developing damage in 33% of patients. CONCLUSION: Having GI damage is associated with a worse prognosis. Patients on a high dose of glucocorticoids are at higher risk of developing GI damage which reinforces the strategy of minimizing glucocorticoids. Oral ulcers appear to decrease the risk of GI damage. © The Author(s) 2021. Published by Oxford University Press on behalf of the British Society for Rheumatology

Comprehensive description of clinical characteristics of a large systemic Lupus Erythematosus Cohort from the Spanish Rheumatology Society Lupus Registry (RELESSER) with emphasis on complete versus incomplete lupus differences

Systemic lupus erythematosus (SLE) is an autoimmune disease characterized by multiple organ involvement and pronounced racial and ethnic heterogeneity. The aims of the present work were (1) to describe the cumulative clinical characteristics of those patients included in the Spanish Rheumatology Society SLE Registry (RELESSER), focusing on the differences between patients who fulfilled the 1997 ACR-SLE criteria versus those with less than 4 criteria (hereafter designated as incomplete SLE (iSLE)) and (2) to compare SLE patient characteristics with those documented in other multicentric SLE registries. RELESSER is a multicenter hospital-based registry, with a collection of data from a large, representative sample of adult patients with SLE (1997 ACR criteria) seen at Spanish rheumatology departments. The registry includes demographic data, comprehensive descriptions of clinical manifestations, as well as information about disease activity and severity, cumulative damage, comorbidities, treatments and mortality, using variables with highly standardized definitions. A total of 4.024 SLE patients (91% with ≥4 ACR criteria) were included. Ninety percent were women with a mean age at diagnosis of 35.4 years and a median duration of disease of 11.0 years. As expected, most SLE manifestations were more frequent in SLE patients than in iSLE ones and every one of the ACR criteria was also associated with SLE condition; this was particularly true of malar rash, oral ulcers and renal disorder. The analysis-adjusted by gender, age at diagnosis, and disease duration-revealed that higher disease activity, damage and SLE severity index are associated with SLE [OR: 1.14; 95% CI: 1.08-1.20 (P < 0.001); 1.29; 95% CI: 1.15-1.44 (P < 0.001); and 2.10; 95% CI: 1.83-2.42 (P < 0.001), respectively]. These results support the hypothesis that iSLE behaves as a relative stable and mild disease. SLE patients from the RELESSER register do not appear to differ substantially from other Caucasian populations and although activity [median SELENA-SLEDA: 2 (IQ: 0-4)], damage [median SLICC/ACR/DI: 1 (IQ: 0-2)], and severity [median KATZ index: 2 (IQ: 1-3)] scores were low, 1 of every 4 deaths was due to SLE activity. RELESSER represents the largest European SLE registry established to date, providing comprehensive, reliable and updated information on SLE in the southern European population

Determining occlusions from space and time image reconstructions

Peer-reviewed paper accepted for presentation at the IEEE Conference on Computer Vision and Pattern Recognition 2016, in Las Vegas, USAInternational audienceThe problem of localizing occlusions between consecutive frames of a video is important but rarely tackled on its own. In most works, it is tightly interleaved with the computation of accurate optical flows, which leads to a delicate chicken-and-egg problem. With this in mind, we propose a novel approach to occlusion detection where visibility or not of a point in next frame is formulated in terms of visual reconstruction. The key issue is now to determine how well a pixel in the first image can be " reconstructed " from co-located colors in the next image. We first exploit this reasoning at the pixel level with a new detection criterion. Contrary to the ubiquitous displaced-frame-difference and forward-backward flow vector matching, the proposed alternative does not critically depend on a pre-computed, dense displacement field, while being shown to be more effective. We then leverage this local modeling within an energy-minimization framework that delivers oc-clusion maps. An easy-to-obtain collection of parametric motion models is exploited within the energy to provide the required level of motion information. Our approach outper-forms state-of-the-art detection methods on the challenging MPI Sintel dataset

Detection and Localization of Anomalous Motion in Video Sequences from Local Histograms of Labeled Affine Flows

International audienceWe propose an original method for detecting and localizing anomalous motion patterns in videos from a camera view-based motion representation perspective. Anomalous motion should be taken in a broad sense, i.e., unexpected, abnormal, singular, irregular, or unusual motion. Identifying distinctive dynamic information at any time point and at any image location in a sequence of images is a key requirement in many situations and applications. The proposed method relies on so-called labeled affine flows (LAF) involving both affine velocity vectors and affine motion classes. At every pixel, a motion class is inferred from the affine motion model selected in a set of candidate models estimated over a collection of windows. Then, the image is subdivided in blocks where motion class histograms weighted by the affine motion vector magnitudes are computed. They are compared blockwise to histograms of normal behaviors with a dedicated distance. More specifically, we introduce the local outlier factor (LOF) to detect anomalous blocks. LOF is a local flexible measure of the relative density of data points in a feature space, here the space of LAF histograms. By thresholding the LOF value, we can detect an anomalous motion pattern in any block at any time instant of the video sequence. The threshold value is automatically set in each block by means of statistical arguments. We report comparative experiments on several real video datasets, demonstrating that our method is highly competitive for the intricate task of detecting different types of anomalous motion in videos. Specifically, we obtain very competitive results on all the tested datasets: 99.2% AUC for UMN, 82.8% AUC for UCSD, and 95.73% accuracy for PETS 2009, at the frame level